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@ Method and apparatus for improving motion compensation in digital video coding. 



7) A novel method and apparatus for use in 
digital video compression provides improved 
block-based motion compensation using the 
global motion of a video frame. A video frame 
comprising a plurality of blocks may be com- 
pressed for transmisston using block motion 
vectors. Motion vectors are generated by block 
matching a block to be coded with a block 
within a reference frame and determining the 
displacement therebetween. To effect block 
matching, motion compensation techniques de- 
fine a search window within one or more refer- 
ence frames within which a displaced block will 
be found. According to the present invention, 
the location of the search window within a 
reference firanne is defined using the global 
motion of the frame. 
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Field of tha Invention 



The present invention relates to tiie field of digital 
video data compression, and in particular video data 
compression using block-based motion compensa- 
tion. 

Background of the Invention 

Block-based motion compensation tecliniques 
including either motion Interpolation or motion predic- 
tion are used in digital image sequence coding to re- 
duce the amount of data that must be transmitted in 
digital video systems. Video sequences typically ex- 
hibit a substantial amount of repetitive image informa- 
tion between consecutive frames. By compensating 
for ttie motion of objects within an image frame, a 
high compression of the video data can be achieved. 

Block-based motion compensation techniques 
initially divide a video frame into several blocks of pic- 
ture elements (pels). These techniques employ mo- 
tion vectors to indicate the movement of these blocks 
between successive frames. It is assumed that each 
pel within a block has the same translational motion. 
These motion vectors, along with error signals, may 
be transmitted to a receiver. The receiver may then 
regenerate the video blocks using the motton vectors 
to locate the corresponding blocks from a previously 
transmitted frame. The error signals are used to ac- 
count for pel differences within each block between 
successive frames. 

Consider, for example, the image of a football in 
a football game video sequence. The football image 
remains substantially identical from frame to frame, 
but its locatton within the video frame changes. If the 
motion vector of the football is known, then the image 
of the football can be reconstructed in a new video 
frame using the football image data from the previous 
frame. As a consequence, instead of transmitting re- 
petitive Image data for each new frame, motion vec- 
tors may be transmitted. An error signal is also trans- 
mitted to account for other differences, for example, 
a change in orientation of the football. 

Motion compensation methods employ a process 
known as motton estimation to generate the motion 
vectors. In motion estimation, a motion vector for a 
particular block within a cunent frame is detemiined 
by matching the image data within the block to the im- 
age data in a displaced block within a reference 
frame. The motion vector represents the difference in 
position between the block from the current frame 
and the displaced block from the reference frame. 
The reference frame may be a previous frame, as in 
motion compensation prediction techniques, ora pre- 
vtous or future frame, as in motion compensatton in- 
terpolatton techniques. 

To effect such block matching, the entire refer- 
ence frame is not ordinarily searched for the dis- 



placed block, as such a search can consume consid- 
erable computatton power and time. Instead, a search 
window is defined within the reference frame. 

In motion estimation, a search window may be 

5 defined in terms of its size in pels and its location 
within the reference frame. In determining search 
window locatton, it is known in the art to center the 
search window on the position of the btock to be 
matched. The method assumes that an image within 

10 a vkleo block is most likely to be found in the vicinity 
of its location in a previous or subsequent frame. The 
search window size is then optimally chosen to be as 
large as the sum of the block size and the maximum 
probable block movement in any direction. Forexam- 

J5 pie, for a 1 5 X 1 5 pel block capable of moving up to 5 
pels in any direction between frames, the maximum 
search window size would be 25 X 25 pels. 

Referring again to the analogy of a football within 
a football game video sequence, the football may, for 

20 discussion purposes, represent a block of video data. 
To find a displaced football in a reference frame, a 
motion estimation method would initially define a 
search window within which it would expect to find the 
football image in the reference frame. The search 

25 window is then scanned for the image with in the win- 
dow most resembling the football. Once the football 
Image Is found, a motion vector may be calculated for 
the football describing its movement between the ref- 
erence frame and the present frame. 

30 Under such methods of locating the search win- 
dow, however, the search window size used for block 
matching limits the range of motion that may be com- 
pensated for. In the 25 x 25 pel search window de- 
fined above, block motion of more than 6 pels cannot 

35 be successfully compensated. If the motion estima- 
tkin method fails to find the proper displaced block, a 
large error signal will have to be transmitted, and the 
advantages of motion compensation will be greatly 
reduced. 

40 To increase the range of motion which may be 
compensated for, it is well known that the search win- 
dow size may be enlarged in order to account for 
greater image movement. The use of larger search 
windows, however, consumes more computation time 

45 and power because there are more candidate dis- 
placed blocks to evaluate to effect block matching. 
For example, if an exhausth/e search is used to effect 
block matching, an increase in size of a 25 x 25 search 
window by a single pel in each direction will increase 

50 the number of candidate displaced blocks by 272- 
252=104 blocks. 

Several attempts have been made to increase 
the range of motion which may be compensated for 
without drastically increasing the computational load. 

55 It is well known in the art to employ search algorithms 
vi/hich are faster than an exhaustive search. These al- 
gorithms include the hierarchical search, the logarith- 
mic search, and conjugate direction, search. Because 
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cuit employing a novel block-based motion com- 
pensation metliod and apparatus according to 
one embodiment of the present invention; 
FIGs. 2A and 2B show a video image frame div- 

5 ided into blocte of pels in accordance with stan- 
dard methods of video data coding techniques; 
FIG 3A shows a functional flow diagram of the 
steps performed by a motion estimation device 
according to the present invention; 

10 FIG. 3B illustrates how a global motion vector 
may be utilized in accordance with the present in- 
vention to define a search window within a refer- 
ence frame to more effectively achieve block 
matching according to the present Invention; and 

15 FIG. 4 shows a functional block diagram of a glo- 
bal motion estimator operable to generate global 
motion vector data for use In the methods and ap- 
paratus of the present inventton. 

20 Detailed Description 



motion estimation techniques using such algorithn^ 
greatly reduce the number of calculations required for 
block matching, they may employ larger search win- 
dows than an exhaustive search routine. 

The faster search techniques, however, are not 
guaranteed to find the proper displaced block. In 
other words, these algorithms may not find the best 
match for a particular block to be coded, which leads 
to coding inefficiency. Furthermore, such search 
techniques require complex image data manipulation 
when compared with the exhaustive search. The 
complex image data manipulation may also computa- 
tionally load the system. 

Summary of the Invention 

The present invention provkles a method of mo- 
tion estimation operable to enable compensation for 
a large range of motion using a novel method of locat- 
ing the search window. Large anraunts of motk>n In 
vkleo are often due to a combination of object motion 
and global motion caused by, for example, camera 
motion. According to the present Inventton, the 
search window is placed In a location that takes into 
account the global motion of the entire video frame, 
thus enabling compensation for greater amounts of 
motion when a component of the motion is caused by 
global motion. 

For example, again consider the image of a mov- 
ing football in a football game video sequence. The 
motion of the football is often due to camera motion 
as well as object motion, which in the process of block 
matching may cause the football image to fall outside 
a reasonably sized search window. If, however, the 
search window is placed in a location in a reference 
frame that takes into account the camera motion, 
then the search window would more likely contain the 
football image, thus facilitating successful compen- 
sation. 

Accordingly, the method of the present invention 
generates a global motion vector from time to time 
and transmits this data to a motion estimation device. 
To perform block matching, the motion estimation de- 
vice uses the global motion vector to define a search 
window in which a displaced block may be found with- 
in one or more reference frames. The motion estima- 
tion device then identifies the displaced block within 
the defined search window. The method of the pres- 
ent invention is compatible with both nKttion compen- 
sation prediction and motion compensation interpola- 
tion techniques. 

The present invention may be utilized with either 
an exhaustive search in block matching, or it may also 
be used with faster search techniques. 

Brief Description of the Drawings 

FIG. 1 showsablockdiagramofavkieocodercir- 



The present Invention provides an improvement 
to current motion estimation techniques used in 
block-based motion compensation by using the glo- 

25 bal motion, caused by, for example, camera motion, 
to assist in defining a search window in the process 
of block matching. A novel video coder circuit 1 00 em- 
ploying motion estimation techniques according to 
the present invention is described below in connec- 

30 Hon with FIG. 1. 

FIG. 1 shows, in block diagram form, vkleo coder 
circuit 1 00 for encoding a sequence of image frames 
represented in digital form using block-based motion 
compensation. The block diagram is shown in simpli- 

35 fled form for purposes of clarity. To this end, various 
aspects of d^ital circuitry are not represented in the 
diagram and should be considered as Inherent to the 
system. For example, a timing and sequence control- 
ler for the various functnnal blocks, as well as imbed- 

40 ded storage elements such as buffers and memories 
are not shown. The details pertaining to the use of 
such digital circuitry in a system according to the 
present invention are well known or will be readily ap- 
parent to one of ordinary skill in the art. 

45 The video coder circuit 100 receives at its input 

101 video data corresponding to temporal frames of 
vkleo. A frame of video comprises a plurality of pic- 
ture elements or pels. The digital representatton of 
the video frame as It appears at the input 1 01 of video 

50 coder circuit 100 comprises pel video data for each 
pel within the frame. The pel video data typically com- 
prises chrominance and luminance values. A video 
frame may be further subdivided into blocks of pels, 
and frames are processed by the video coder circuit 

55 1 00 on a block by block basis. 

The video coder circuit 1 00 produces at its output 

102 a compressed and encoded digital signal in a bit 
stream for subsequent storage or transmisston over 



3 



5 



EP 0 652 678 A2 



6 



a transmission medwm, not shown. The compressed 
and encoded digital signal may be suitably coded ac- 
cording to the IWPEG1, MPEG2 or any other video 
data transmission standard that Is designed specifi- 
cally to work with data compression techniques in- 
cluding block-based motion compensation prediction 
and interpolation. See, for example, ANSI Standard 
ISO-IEC DtS 11172, "Coding of Moving Pictures and 
Associated Audio for Digital Storage Media For Up to 
About 1.5 Mbit/sec." 

The vkleo coder circuit 100 illustrated in FIG. 1 
compresses and encodes data employing motion 
compensation prediction techniques. In processing 
the blocks of a video frame, the circuit 100 may either 
encode a particular video data block directly or en- 
code the block using motion compensation predic- 
tion. Video blocks that cannot be successfully com- 
pensated are encoded directly. 

The f unctk>nal elements in the video coder circuit 
1 00 are interconnected as follows. A video frame stor- 
age unit 103 is operably connected to transfer both 
video data and position data corresponding to a new 
block of video to be coded ("new block") to a motion 
estimator 104, and to transfer the same new block 
vkJeo data to both a subtraction node 1 05 and a cod- 
ing device 107. The video frame storage unit 103 in- 
cludes a digital memory or buffer which stores both 
video data con-esponding to one or more frames and 
is operable to generate and store block position data 
for the video blocks within each stored frame. 

The motion estimator 104 is operably connected 
to receive block video data from the video frame stor- 
age unit 1 03 and global motion data from a global mo- 
tion estimator 106. The motion estimator 104 com- 
prises a processing means such as a microprocessor 
and related memory circuitry which Is operable both 
to effect block matching using global motion data and 
to generate motion vectors. The motion estimator 104 
further includes motion compensation means for pro- 
viding displaced block data to the subtraction node 
105. According to one embodiment of the invention, 
the processing means within the motion estimator 
104 is programmed to execute the block matching 
and motion vector generation consistent with theflow 
chart 300 described below in connection with FIG. 3A. 
The motion estimator 104 is further connected to 
transfer block video data to the subtraction node 105 
and to provkle motion vector data to both a coding de- 
vice 107 and the global motton estimator 106. 

The motion estimator 104 is further connected to 
a reference frame storage buffer 108 and a new 
frame storage buffer 1 09. The newf rame storage buf- 
fer 109 is connected between an additton node 110 
and the reference frame storage buffer 108. Both the 
new frame storage buffer 109 and the reference 
frame storage buffer 108 have sufficient storage 
space to hold the video data corresponding to at least 
one video frame. 



The subtraction node 105 Is operably connected 
to receive new block video data from the video fratm 
storage unit 103 and to receive displaced block video 
data from the motion estimator 104. The subtraction 

5 node 105 Is further connected to the coding device 
107. The subtraction node 1 05 comprises a data stor- 
age means, such as a buffer and means for generat- 
ing an error signal representing the difference in pel 
luminance and chrominance between two video data 

10 blocks received from the rrtotion estimator 104 and 
the vWeo frame storage unit 103. 

The coding device 107 is operably connected to 
recewe the error signal from the subtraction node 
105, motion vector data from the motion estimator 

15 1 04 and block video data from the video frame stor- 
age unit 103. The coding device 107 is further con- 
nected to provide an error signal to the addition node 
110. and includes the output 1(S fortransmitting cod- 
ed video data in the form of a bit stream. The coding 

20 device 1 07 comprises a means for encoding and de- 
coding video data using the MPEG1, MPEG2 or other 
coding standard for transmitting a coded bit stream. 
Devices capable of coding in these standards are 
known in the art. For more infonmation on such coding 

25 methods and devices, see ANSI Standard ISO-IEC, 
above, incorporated by reference herein. 

The global motion estimator 106 is connected to 
receive motion vectors from the rrratton estimator 104 
and to provide the global motion vector to motkin es- 
se timator 104. The global motion estimator comprises 
the processing means and related circuitry operable 
to store a plurality of motion vectors and to compute 
therefrom a global motton vector that is representa- 
tive of a common motion component within an entire 

35 video frame. Although the global motion estimator 
1 06 is illustrated as a separate element, the functions 
performed by the global motion estimator processing 
means may be performed by either a dedteated mi- 
croprocessor or a microprocessor Included within the 

40 system to perform other control and operational func- 
tions. To achieve its desired functions, the global mo- 
tion estimator 106 may suitably be programmed con- 
sistently with the flow chart 400 discussed below in 
connection with FIG. 4. 

45 The addition node 110 is connected to receive 
block vkieo data from the motion estimator 104 and 
error signal from the coding devtee 107. TTie addition 
node 110 comprises means for combining a block of 
vkleo data with an error signal to produce a compen- 

50 sated block of video data. The addition node provides 
the compensated block to new frame memory buffer 
109. 

The video coder circuit 100 described above op- 
erates in the following manner to process and encode 
55 a new video frame, one block at a time, using block- 
based motion compensation with global motion esti- 
mation. A video frame to be coded is first dwided into 
blocks. Each block is then encoded either directly or 
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using motion compensation prediction until the entire 
frame is encoded. The decision of whether or not to 
encode using motion compensation is made by the 
control circuitry using predetenmined criteria, typical- 
ly dependent on the size of the error signal produced 
during motion compensation. The next frame is there- 
after processed in the same manner. 

It should be noted that the various elements dis- 
cussed below exchange data using digital signals, the 
generation of which is known to those of ordinary skill. 
For purposes of clarity, it will be understood that when 
any element is said to provide or receive data, for ex- 
ample, representing video btodcs or motion vectors, 
to or from another element, the element Is actually 
providing or receiving digital signals containing the 
data. 

In order to process a video frame one block at a 
time, the control circuitry directs that each frame, pri- 
or to processing, be divided into blocks of video data, 
DB|,t, where i is the block number within a frame cor- 
responding to a time t. Initially, video data defining a 
video frame Ft corresponding to a time t in a video se- 
quence is stored in a buffer or other storage means 
within the video frame storage unit 1 03. The time t ac- 
tually represents a sequential frame number such 
that, for example, F(.i is the frame immediately pre- 
ceding Ft, and Ft^t is the frame immediately subse- 
quent to Ff The control circuitry directs that each 
frame be divided into a number of blocks per row m, 
and a number of blocks per column n, so that the 
frame has total number of blocks, NB, or in other 
words, m X n = NB blocks. The control circuitry may 
be user controllable to select the number of blocks per 
row m and the number of blocks per column n. Once 
the number of blocks is determined, the video frame 
storage unit 103 then obtains and stores block posi- 
tion values BP|,t for each block within one vWeo frame. 

For the present embodiment of the invention, the 
block position value may suitably be defined as the 
pel position value of the pel in the upper left hand cor- 
ner of each block. The pel position value constitutes 
the position within the two-dimensional array of pels 
comprising the video frame. The pel position may 
suitably be provided by the contnal circuitry. The vid- 
eo frame storage unit 103 may determine the block 
position for each block i, BPy = (x,y), using any appro- 
priate method. 

For example, FIG. 2A illustrates the frame Ft 201 , 
which is divided into 16 rows and 16 columns of 
blocks. Block 202 represents data block DBi9,t in 
frame 201 wherein the first block 203 is DBo.t and the 
block i-values are counted from left to right and top to 
bottom. Block 202, as shown in FIG. 2B comprises a 
15X15 array of pels, one of which is exemplified by 
upper left corner pel 204. The pel position value for 
pel 204 is (45,15), corresponding to its x,y coordin- 
ates. These coordinates are determined as follows. 
Pel 204 Is the upper left hand pel of block 202. Block 



202 is the 19th block in frame F, 201, and it is in the 
fourth column and second row of frame F, 201 . X-pels 
0-14 are in blod<s in the same column as block 203, 
x-pels 15-29 are in blocks in the second column, x- 

5 pels 30-44 are in blocks in the third column, and x- 
pels 45-59 are in blocks in the fourth column, which 
is the column block 202 is in. Thus, the left most pels 
of block 202 will have an x-coordinate of 45. LI kewise, 
y-pels 0-14 are in blocks in the same column of block 

10 203, and y-pels 15-29 are in blocks in the next col- 
umn, which is the column block 202 is in. Thus, the up- 
per most pels of block 202 will have a y-coordlnate of 
15. The block position value of block 202, as dis- 
cussed above, is the pel position value for its upper 

}5 left hand corner pel 204. Therefore, BPig,, = (45, 1 5). 
Once the data blocks and their respective block 
position values are defined, the control circuitry di- 
rects that each block DB|,, be processed, block by 
block. Returning to FIG. 1, the control circuitry in- 

20 structs the vkieo frame storage unit 103 to provide a 
new block of video data, DB|,t and the con^ponding 
block positton data, BP|,t to the motion estimator 104. 
The new block video data DB|,t is further transmitted 
to the subtractton node 1 05 and stored in a buffer 

25 therein. 

The motion estimator 104 then effects block 
matching in order to matoh DB|,t with a data block. 
MB,ef, from the video frame stored in the reference 
frame storage buffer 108. The storage buffer 108 

30 contains at least one reference vMeo frame. Because 
the block is to be encoded using motion compensa- 
tion predkstton in the present embodiment, the refer- 
ence frame comprises the previous video frame, Ft. 
In motion compensation prediction, the location of 

35 the displaced block represents the location of the vid- 
eo image defined by the new block within the previ- 
ous frame. 

If, however, the new block is to be encoded using 
motion compensation interpolation techniques, the 
40 motion estimator 1 04 locates MB,^ within a plurality 
of reference frames stored in the reference frame 
storage buffer 109. The plurality of reference frames 
may correspond to both previous and/or future video 
frames depending on the chosen motion compensa- 
45 tion technique. In such cases, the motion estimator 
1 04 performs the block matching step for the plurality 
of reference frames. 

The motion estimator 104 is operable to effect 
block matehing for each reference frame in the follow- 
so ing manner. The motion estimator 1 04 first defines a 
search window within the reference frame in which 
the displaced block may be expected to be found. Ac- 
cording to the present invention, the motion estimator 
104 uses a global motion vector in defining the loca- 
55 tion of the search window. The global motion vector 
is representative of motion common to all the blocks 
within the video frame F,, and is from time to time pro- 
vided by the global motion estimator 106 as dis- 
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cussed in detail further below. The defined search 
window is thereafter searched until the displaced 
block MB^ for the new block DBy is found. 

Once a displaced block MBre, is found, the motion 
estimator 104 determines the motion vector MVj.tjt s 
con-esponding to the new block. The value dt indi- 
cates the frame number difference between F, and 
the reference frame. For motion prediction, the refer- 
ence frame wilt typically be the previous frame, and 
therefore dt = 1. The motion vector MV|^ comprises io 
a two-dimensk>nal value (dx,dy) representing the hor- 
izontal and vertical displacement between the new 
block DB|,t and the displaced block MB^ei. The two-di- 
menstonal motion vector value (dx,dy) may suitably 
be determined by calculating the difference between is 
the new block position value BP|,i and the pel position 
value of the pel in the upper left corner of the dis- 
placed block MBref. For example, referring to Fig. 2, 
the block 202 (DBi9,0 has a block position value BP1 g,t 
= (45,15). If it is further assumed that a displaced 20 
btodk has been identified as conresponding to block 
202 and that the pel position value of the upper left 
hand corner pel of the displaced block MBref is 
(50,21), then the motion vector MVig^i = (dx.dy) = 

(+5,+6). 25 

Afunctional block diagram of an exemplary block 
matching and motion vector generating method which 
may be employed by the motion estimator 104 to ach- 
ieve the foregoing Is discussed further below in con- 
nection with FIG. 3A. 30 

The displaced block video data MB,Bf in the mo- 
tion compensator 104 is then transmitted to the sub- 
traction node 105. The displaced block video data is 
also transmitted to the addition node 11 0 for purposes 
discussed further below. In the subtraction node 105, 35 
the displaced block video data MBref, comprising both 
luminance and chrominance data, is subtracted on a 
pel by pel basis from the new block video data DBy 
received and stored from the video frame storage unit 
103. The resulting error signal ERROR represents 40 
the difference between the displaced block MB^f and 
the new block DB|,,. The signal ERROR is then provid- 
ed to the coding device 107. 

The motion estimator 104 provides the motion 
vector MV|,t)t to the coding device 107. The motion 45 
vector is also provided to the global motion estimator 
1 06 for purposes discussed further below. The coding 
device 107 comprises digital circuitry, which may in- 
clude a microprocessor, that is capable of producing 
digital signals according to the MPEG1, MPEG2 or 50 
other standard to the video coder output 102. 

The coding device 107 also receives the new 
block video data DBy from the video frame storage 
buffer 103. The control circuitry then determines 
whether the motion compensated version of the 55 
block, in other words, the motion vector and the signal 
ERROR, should be encoded and transmitted. If not, 
the control circuitry instructs the coding device 107 



to encode the new block video data D6|,t. If, however, 
the motkin compensated blodc is to be transmitted, 
the control circuitry instructs the coding device 107 
to encode for transmission the motion vector MV|_dt 
and the signal ERROR. The encoded data is then pro- 
vided to the circuit output 102 for transmission to one 
or more receiving devices, not shown. In either case, 
the encoded data is transmitted in a bit stream along 
with appropriate signals to indicate whether the mo- 
tion compensated version of the block has been 
transmitted. 

After transmission, the new block DB|,t is stored 
in order to build the current frame in new frame stor- 
age buffer 109. If the nfMtion compensated version of 
the block was transmitted, the addition node 110 will 
receive a decoded version of the coded error signal 
from the coding device 107 and will add the error sig- 
nal to MBref which is received from the motion estima- 
tor 104. The resulting block, called a compensation 
block, closely resembles the new block DB|,t. The 
compensatton block is placed inside the new frame 
buffer 109 In a positton con^sponding to the new 
block position value, BP|,,. 

If, however, the motion compensated version of 
the block was not transmitted, the new block video 
data DB|,t may be placed directly into the new frame 
buffer 109 from the motion estimator 1 04. In this man- 
ner, the frame Ft is built within the new frame storage 
buffer 109 which will eventually be placed in the ref- 
erence frame storage buffer 108. 

Concurrent with the operation of the coding de- 
vk:e 107 and the additton node 110, the global motion 
estimator 1 06 receives the motion vector MVi,* The 
motion vector MVi^ is received into a buffer, stack or 
memory within the global motion estimator 106. 

The process performed by video coder circuit 
100, as outlined above, is repeated for each block 
DB|,t of each frame until an entire video frame (i=NB) 
has been encoded by the video coder circuit 100. 
Once the entire frame Ft has been processed, the 
control circuitry will cause the new frame buffer 109 
to transfer the video data corresponding to the frame 
F, to the reference frame storage buffer 108 in order 
to process the next frame conresponding to the time 
t+1. 

Furthermore, before the next frame is processed, 
the global rration estimator 106 will generate and 
transmit a new global motk>n vector to the motion es- 
timator 104. The global motion estimator 106 gener- 
ates the global motion vector using the motion vec- 
tors stored during the processing of F,. The global mo- 
tion vector approximates the component of motion 
common to every block in a frame of a video se- 
quence which may be due to camera motion. The mo- 
tion estimator 104 receives and stores the global mo- 
tion vector GMV prior to receiving the new video data 
block for the first block of the next frame (DBo,tn). The 
global motion estimator may be programmed to exe- 
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cute the above functions as dtecussed below in con- 
nection witli FIG. 4 

It will be understood that the video coder circuit 
1 00 described in connection with FIG. 1 is exemplary. 
The global motion estimation method and apparatus 5 
of the present invention may be utilized in any block- 
based video motion compensation device that per- 
forms biocl< matching to compute motion vectors be- 
tween a block to be coded and a displaced blod< with- 
in one or more reference frames, by first defining a io 
search window In the reference frames. For example, 
U.S. Pat. No. 5,151,784 Incorporated herein by refer- 
ence illustrates a video coder circuit incorporating 
such a motion compensation device that Is operable 
to employ both motion compensation prediction and i5 
motion compensation interpolation for coding block 
based video data. The method according to the pres- 
ent invention may readily be adapted for use therein. 

FIG. Sfii illustrates a functional flow chart for a 
motion compensation device that employs a global 20 
motion vector to effect block matching according to 
the present invention. The flow chart may suitably be 
executed by the motion estimator 104 in FIG. 1. The 
functionality of the flow chart, however, is designed 
to be compatible with alternath/e vkleo coder circuits 25 
employing both motion compensation predictbn and 
motion compensation interpolation. The processing 
means and memory nraans (processor) within the 
motton estimator execute a program perfonming the 
steps in FIG. 3A to effect block matching and motion 30 
vector generatton. 

The processor first receives new block video data 
DBi,t for a block i within a frame F, in step 302. Next, 
in step 310, the search window Aref within the refer- 
ence frame is defined. Aj^t is video data representing 35 
the portion of the reference frame within which a dis- 
placed block MBref for DBy may be located. The loca- 
tion of Af,f within a the reference frame is defined by 
first centering A^ef on the new block position BPy, and 
then displacing A^ef by an amount defined by the stor- 40 
ed global motion vector GM. 

It should be noted that the global motion vector 
GM is normalized to represent the global motion be- 
tween the current frame and the immediately preced- 
ing frame, F^,. As a consequence, if the reference 45 
frame is not the imntediately preceding frame, the 
global vector should be scaled accordingly. Scaling 
may suitably be linear. For example, if the reference 
frame corresponds to a time t-2, then the global mo- 
tion vector should be doubled before determining the 50 
location of Afef. Likewise, in motion compensation in- 
terpolation, the reference frame may be a future 
frame, in which case the global motion vector should 
be inverted. By storing and subsequently scaling a 
normalized global motion vector,, the processor may 55 
suitably apply the global motion vector to block 
matching operations in both interpolated and predict- 
ed modes. 



The size of the search window Anf shouki prefer- 
ably be large enough to ensure induston of the dis- 
placed block, but small enough so that the blodc 
matching function does not unduly load the process- 
ing means of the motion estimator. The size of search 
window Afef is a predetermined number which takes 
into account the maximum object motion between 
video frames and may vary depending upon the ap- 
plicatton, as well as other factors. In the context of a 
particular applicatton, the detenmination of a proper 
size will be apparent to one skilled in the art 

FIG. 3B further Illustrates the novel method of 
defining the search window A^ as described in con- 
nection with step 310. In FIG. 3B, the given block 354 
from the frame Ft, effects block matching to identify 
displaced block 351 from the reference frame, which 
may suitably be the previous frame. 

The image in a data block DB|,t is assumed to 
have moved from the position of block 351 in the ref- 
erence frame to the position of DB^ shown by the 
block 354. The movement of the image In DB|,t is due 
In substantial part to camera movement or some other 
source of global motion. The reference frame 352 has 
the same dimensions as the video frame to be coded 
from which DB,,, originated. 

The search window A^f is located by first center- 
ing the search window on the position of the block 
position for the block 354, as illustrated by centered 
window 356. The search window is then displaced by 
the global motion vector 355, as Illustrated by dis- 
placed window 353. The global motion vector 355 Is 
representatwe of the component of motton, which 
may be caused, for example, by camera movement, 
that is common to all blocks within a frame. The global 
motion vector 355 may suitably be provided from time 
to time by the methods described below In connection 
with FIG. 4. 

The use of global motion in defining the search 
window location provides a dear improvement over 
the prior art Refenring to FIG. 38, the prior art method 
for locating the search window consists essentially of 
centering the search window around the positton of 
the new block 354, as illustrated by centered window 
356. See, for example, Jain, et al, "Displacement 
Measurement and its Application in Interframe Image 
Coding," 29 IEEE Transactions on Communications 
pp. 1799-1808 (1981) incorporated herein by refer- 
ence. It can be seen that the centered search window 
356 does not Include the displaced block 351 . If the 
displaced block is not found, the motion estimator 
cannot effectively aid video compression. 

The centered search window 356 could, of 
course, be enlarged to encompass displaced block 
351. As discussed above, however, enlarging the 
search window can greatly increase the number of 
calculations required for block matching. Further- 
more, because many video sequences have little or 
no camera motion, It would be wasteful to enlarge the 
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search window when a smaller search window is ade- 
quate in still sequences. By using global motion, the 
search window can remain a reasonable size yet still 
encompass the displaced block In large camera 
movement sequences. Consequently, because the s 
use of global motion allows for a larger range of mo- 
tion when camera or global motion is present, it pro- 
vkJes a substantial improvement to motion compen- 
sation techniques. 

Returning to FIG. 3A, once the size and location io 
of the search window A^f is chosen, the motion esti- 
mator identifies a displaced biocic MB^ within the 
search window using predetermined criteria in steps 
312, 314 and 316. The identification of a displaced 
block within a reference frame Is well known in the art. 15 
See, for example, Jain, et al, above, at 1799-1808. 
The displaced block is a block of data within the 
search window having the least image difference 
when compared to the DBi,,. in step 312, the block 
DB|,t Is compared with several candidate displaced 20 
blocks in the search window in order to determine the 
candidate block having the least image difference. In 
step 314, the data for the candidate blocks is retrieved 
as needed from a means for storing the reference 
frame, for example, the reference frame storage buf- 25 
fer 1 08 of FIG. 1 . The image difference may suitably 
constitute the mean-absolute-difference or the 
mean-squared-enror between the total pel luminance 
value for the new block DB|,t and the total pel lumi- 
nance value for each candidate displaced block. For 30 
more information on such block matching techniques, 
see, for example, Jain, et al, above, at 1800; Sriniva- 
san, et al, "Predictive Coding Based on Efficient Mo- 
tion Estimation," Proc. Int'l Conf. Communications 
pp. 521-26 (Amsterdam, May 14-17, 1984). 35 

The choice of candidate blocks within the search 
window for which such image comparisons are made 
in step 312 is dictated by a predetermined search 
methodology. The search methodology may suitably 
be an exhaustive search, which compares every de- 40 
finable block within the search window with the new 
block. The exhaustive search is often preferable be- 
cause it is guaranteed to find the block In the search 
area having the minimum error. The search method- 
ology may alternatively be a logarithmic search, a hi- 45 
erarchical search or other search method which is 
known in the art. For more information on these alter- 
native search methodologies, see, for example, Jain, 
et al, above; Srinh/asan, et al, above; and Bierling, 
"Displacement Estimation by Hierarchical Block- so 
matching," Proc. Visual Communications & Image 
Processing "88, (SPIE. November 9-11, 1988). 

Once the displaced block MBref is identified, the 
motion estimator 1 04 derives the motion vector MV|,d, 
for the new block DB,,, in step 31 8. The motion vector 55 
is defined as the displacement between the new 
block position BP,,, and the pel position of the upper 
left hand corner pel, or the f h-st pel, of the displaced 



block, MBref. The dfeplaced block is thereafter pro- 
vided in step 320 to other elements in the coding cir- 
cuit, for example, the subtraction node 105 and addi- 
tion node 110 as illustrated in FIG. 1 above. In step 
322, the motion vector MVi.^ is provided to the global 
motion estimator and other elements in the coding 
cffcuit, for example, the coding device from FIG. 1. 

While motion compensation prediction employs 
only one reference frame, motbn compensation in- 
terpolatton performs block matching using more than 
one reference frame. As a consequence, in step 324, 
the processor must determine if there is another ref- 
erence frame with which to perform block matching. 
If In step 324 the processor determines that another 
block matching step must be performed, the proces- 
sor executes step 326. In step 326, the processor in- 
structs the reference frame storage means to make 
the next reference frame available for block match- 
ing. The processor then returns to step 310 to define 
a search window wtthin the new reference frame. If, 
however, the answer to step 324 is no, the processor 
executes step 328. 

In step 328, the processor determines whether 
there are more new data blocks within the current 
frame to be compensated. This information may suit- 
ably be provided by the control circuitry. If no, the 
processor retrieves a new global motion vector GMV 
from the global motion estimator in step 330. The glo- 
bal motion estimator may suitably operate according 
to the method discussed below in connection with 
FIG. 4. In step 332, the new global motion vector is 
stored as GM for use within the motion estimator 104 
for processing the next video frame, which In motion 
compensation prediction would typically be F^,,. The 
motion estimator 1 04 thereafter executes step 302. If, 
however, the answer in step 328 is yes, the motion es- 
timator 104 proceeds directly to step 302 to receive 
further blocks from the current frame. 

FIG. 4 shows an exemplary functional flow dia- 
gram of the novel global motion estimation program 
according to the present invention. The program is 
executed by processing means within a global motion 
estimator such as the global motion estimator 106 
that is discussed above in connection with FIG. 1. The 
global motion estimator may suitably be a digital mi- 
croprocessor circuit with associated memory and buf- 
fer configurations capable of executing the software 
as well as receiving and transmitting the appropriate 
data from and to other components. Such circuitry is 
well known in the art. In one embodiment, the micro- 
processor required to execute the program may be a 
dedicated microprocessor. However, because the 
global motion estimation computation is relatively 
simple and occurs only once per video frame, the pro- 
gram may be executed by a shared microprocessor, 
including, for example, one used for timing and con- 
trol of the overall coder circuit 

Essentially, the global motion estimator gener- 
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ates a global motion vector for a video frame by esti- 
mating the component of nratton common to every 
block In the entire frame. This motion component is 
typically caused by camera motion or panning. The 
global motion estimator according to the present in- 
vention uses the block motion vectors associated with 
the previous frame to calculate the present global mo- 
tion component 

A global motion estimator embodiment that exe- 
cutes the steps illustrated In FIG. 4 operates by first 
defining a common motion vector by taking the aver- 
age of the motion vectors stored during the coding of 
a previous frame. A new global motion vector may 
then be provided to a motion estimator programmed 
to operate as discussed above in reference to FIG. 
3A. Other methods for estimating the global motion 
may be readily employed by one of ordinary skill in 
the art. 

In step 401 , motion vector data defining a motion 
vector, MV|,« for a block i, between a frame and a ref- 
erence frame dt frames apart, is received from a 
block-based motion estimator. The motton estimator 
may suitably operate In accordance with FIG. 3A 
above to effect block matching and motion vector 
generation. In step 401 , the received motion vector is 
also normalized so that it represents the motion be- 
tween consecutive frames. A normalized nration vec- 
tor is given by equation MV|,dt/dt where dt is the cur- 
rent frame number minus the reference frame nunn- 
ber. In step 402, the motion vector, MV|,«, is added to 
a motion vector total, MVT, for the frame. Alternative- 
ly, the motton vector, MV|,dt. may be stored in a mem- 
ory In order to facilitate alternative global motion vec- 
tor computation methods, as discussed further be- 
low. In step 403, a counter C is advanced to track the 
number of motion vectors received for the current 
frame. 

In step 404, the global motion estimator may re- 
turn to receive another motton vector for the current 

block i. If the coder circuit employs motion compen- 
sation interpolation, multiple motion vectors, typically 
two, are generated during the processing of each new 
block of video data. Control circuitry typically pro- 
vides this information to the global motion estimator 
so that the appropriate decision in step 404 can be 
made. If there are no more motion vectors for the 
block i, step 405 is executed. 

In step 405, the global motion estimator determi- 
nes whether there is another block of the current vid- 
eo frame to be motion compensated, in other words, 
whether another motion vector is expected for the 
frame F,. If another motion vector is expected for the 
current frame, the program returns to step 401 and 
waits for the next motion vector, MVi,^ from the mo- 
tion estimation device. If, however, another motion 
vector is not expected for the frame, the global motion 
estimator executes step 406. 

In step 406 the common motion vector is calcu- 



lated for the current frame. The common motion vec- 
tor may be the average nratton vector determined by 
divWing the motion vector total by the number of vec- 
tors, C, received for the frame, or in other words, 

5 MVT/C. Alternatively, the common motion vector may 
suitably be the median motion vector or some other 
value as discussed below. 

In step 407, the global motion vector GMV is gen- 
erated for use in motton estimating the next frame, 

10 which for motion compensatton prediction would be 
the frame F(»v Because the motion vectors are nor- 
malized as they are received, the resulting global mo- 
tion vector will also already be normalized. As dis- 
cussed above, the global motion vector may then be 

15 scaled in the motion estimator to allow for block 
matching therein with respect to any reference frame. 
The global motion vector GMV may suitably be the 
common motion vector calculated in step 406. It 
should be noted, however, that the use of multiple 

20 common motion vectors to generate a global motion 
vector may allow more sophisticated methods of es- 
timating the global motton. 

In step 408, the global motion vector is provided 
to a connection to the motion estimator which is op- 

25 erable to perform block-based motion compensation. 
The motion estimator 104 may suitably perform the 
method discussed above In connection with FIG. 3A. 

In the functional block 409, the counter C and mo- 
tion vector total MVT of the global motion estimator 

30 are reinitialized to prepare forthe global motion vector 
calculation for the next video frame. 

In an alternath/e embodiment of the global motion 
estimation method, the median motion vector from a 
plurality of stored motion vectors may provide the 

35 common motion vector in step 406. In such an em- 
bodiment, step 402 of FIG. 4 would include the oper- 
atton of storing the motion vector MV|^. Step 406 
would then include the step of analyzing the several 
stored motion vectors to determine the median. The 

40 foregoing steps may be readily implemented. 

In yet another alternative embodiment, the global 
motion vector may suitably be determined from a his- 
togram. Under this method, the motion vector occur- 
ring with the most repetition within the processing of 

45 a frame would constitute the global motion vector 
To implement this embodiment, step 402 would 
include the step of updating a histogram of a table of 
possible motton vector values. For example, if MV|^ 
= (0,+3), the occurrence index for (0,+3) within the ta- 

50 ble of possible nmtion vector values is incremented by 
one. The populatton of possible motion vectors may 
suitably be created for each new frame, whereby the 
value of motion vector MV,,* is added to the popula- 
tion unless it is already represented. For example, if 

55 MV|,rt = (0,+3), and no occurrence index for that value 
exists, then {0,+3) is added and its index is increment- 
ed to one. 

Additionally, step 405 would include the step of 
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scanning the table for the largest occurrence Index. 
The motion vector value occurring the most is then 
determined to be the common motion vector. The pro- 
gramming necessarily to achieve the foregoing may 
be readily accomplished. s 

The usefulness of this embodiment is clear. It 
may assumed that most objects (and blocks) within a 
video frame do not exhibit independent movement. 
For example, background objects in most scenes are 
relatively static. However, if camera motion or global io 
motion is present, these otherwise static objects will 
all have the same motion vector. Because indepen- 
dently moving objects having the exact same motion 
vector will seldom outnumber the static objects, the 
motion vector having the most occunrences within a 15 
video frame is assumed to represent the global mo- 
tion. 

It is to be understood that the above-described 
arrangements of the invention are merely illustratwe. 
Other arrangements may be devised by those skilled 20 
in the art which will embody the principles of the In- 
vention and fall within the spirit and scope thereof. 



Claims 25 

1. A method of converting a block of video data 
which is one of a plurality of blocks defining a 
frame of video data, into a compressed encoded 
digital bit stream for transmission over a trans- 30 
mission medium, said method comprising: 

a) generating a global motton vector 

b) generating both video data and position 
data conresponding to the block of video data 

to be coded; and 35 

c) effecting block matching by using predeter- 
mined criteria to identify a displaced block 
corresponding to the block of vWeo data to be 
coded, the displaced block having a location 
within a reference frame, said block matching 40 
step further comprising 

defining a search window within the reference 
frame using the global motion vector. 

2. The method of claim 1 further comprising the 45 
steps: 

d) generating a motkin vector defined by the 
block position data and the locatton of the dis- 
placed block within the reference frame; 

e) storing said motion vector; so 

f) encoding said motion vector into a bit 
stream for transmission; and 

g) repeating steps b) through f) for a plurality 
of blocks to be encoded, said plurality of 
blocks constituting at least a portion of one ss 
frame of video data. 



erating the global motion vector further compris- 
es the step of utilizing a plurality of the previously 
stored motton vectors from said step (e) to gen- 
erate the global motion vector and further com- 
prising the step of: 

h) repeating steps a) through g) for a plur- 
ality of frames of video data. 

4. The method of daim 3 wherein the plurality of 
previously stored motion vectors comprise mo- 
tion vectors stored during the encoding of a frame 
of video data. 

5. The method of claim 3 wherein the step of gen- 
erating the global motion vector utilizing a plural- 
ity of the previously stored motion vectors further 
comprises utilizing the mean of the plurality of the 
previously stored motton vectors to generate the 
global motion vector. 

6. The method of claim 3 wherein the step of gen- 
erating the global motion vector utilizing a plural- 
ity of the prevtously stored motion vectors further 
comprises utilizing the median of the plurality of 
the previously stored motion vectors to generate 
the global motion vector. 

7. The method of claim 3 wherein the step of gen- 
erating the global motion vector utilizing a plural- 
ity of the prevtously stored motion vectors further 
comprises utilizing the motion vector occurring 
with the most repetition within the plurality of the 
previously stored motton vectors to generate the 
global motion vector. 

8. An apparatus for determining the location of a 
displaced block of video data within a reference 
frame corresponding to a block of video data 
where the displaced block is identified from a 
plurality of candidate displaced blocks using pre- 
determined criteria, the apparatus comprising: 

a) a motton estimator comprising a process- 
ing means and memory means, the process- 
ing means operable to determine the location 
of a displaced block by defining a search win- 
dow within the reference frame using a global 
motion vector, identifying a plurality of candi- 
date displaced blocks having a location with 
the search window, and comparing each can- 
didate displaced blocks with the block of video 
date; and 

b) a global motion estimator coupled to the 
motion estimator, comprising processing 
means and storage means, said processing 
means operable to generate global motion 
vectors. 



3. The method of daim 2 wherein the step of gen- 
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9. The apparatus of daim 8 wherein the motton es- 
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timator further comprises means for effecting 
motion compensation prediction. 

10. Ttie apparatus of claim 8 wherein the motion es- 
timator further comprises means for effecting 5 
motion compensation interpolation. 

11. A video coder circuit for converting a plurality of 
frames of video data comprising a plurality of 
pels, each pel having pel video data, a plurality of io 
pels further defining a block, into a compressed 
encoded digital bit stream for transmission over 

a transmission medium, said video coder circuit 
comprising: 

a) a video frame storage unit operable to gen- is 
erate a block of video to be coded from a 
frame of video data, said blod< having a pos- 
ition within the frame of video data; 

b) a motion estimation means connected to 

the video frame storage unit for receiving the 20 
video data corresponding to a block of video 
to be coded, said motkin estimation means 
operable to effect block matching between a 
block of video to be coded and a block within 
a reference frame to Identify a displaced 25 
block having a location within the reference 
frame, and to generate a motion vector de- 
fined by the position of the block to be coded 
and the location of the displaced block within 
the reference frame, said motion estimation so 
means further operable to effect block match- 
ing using a global motion vector; 

c) a first buffer for storing video data corre- 
sponding to one or more reference frames 
connected to the motion estimation means for 35 
providing reference frame video data thereto; 

d) a second buffer for storing video data cor- 
responding to a current frame connected to 
the first buffer for providing current frame vid- 
eo date thereto and connected to the motion 40 
estimation means; 

e) a subtractton node connected to the video 
frame storage unit for receiving block video 
data and connected to the motion estimation 
means for receiving displaced block video 45 
data, said subtraction node operable to gen- 
erate an enror signal representative of the dif- 
ference in pel video data between two blocks 

of video data; and 

f) a coding device connected to the subtree- so 
tion node for receiving an enror signal and con- 
nected to the motton estimation means for re- 
ceiving a motton vector, saki coding device 
operable to encode video data from a block 
error signal and motion vector data Into a bit 55 
stream. 

12. The system of daim 11 further comprising a glo- 

11 



bal motton estimation means connected to the 
motion estimation means for provkling the global 
motion vector thereto, saW global motion estima- 
tion means operable to generate global motion 
vector data representatwe of the global motion 
between video frames. 

13. The system of claim 12wherein the global motion 
estimation means is operably connected to re- 
ceive motion vectors from the nmtion estimation 
means and is further operable to generate a glo- 
bal nrration vector from a plurality of received mo- 
tion vectors. 

14. The video coder circuit of claim 11 wherein the 
motion estimation means further comprises 
means for effecting motion compensation predic- 
tion. 

15. The video coder circuit of daim 11 wherein the 
motion estimation means further comprises 
means for effecting motion compensatton inter- 
polation. 

16. A method of defining a search window within a 
stored reference video frame for use in block 
matching in a block-based motion estimation de- 
vice, the block-based motion estimation device 
being operable to effect block matching and gen- 
erate motton vectors representing the displace- 
ment between a new video data block and a dis- 
placed block within the reference video frame, 
the search window definition method comprising: 

a) generating a motion vectorforthe new date 
block from a frame of video date comprising 
a plurality of date blocks; 

b) storing said motion vector In a memory; 

c) repeating steps a) and b) until a plurality of 
motion vectors for a plurality of said date 
blocks within said frame of vkieo date have 
been generated and stored; 

d) generating a global motion vector from the 
plurality of the stored motion vectors; 

e) Identifying a video date block having a lo- 
catton within a frame of video data to be cod- 
ed; and 

f) defining a search window having a location 
within a stored reference video frame wherein 
the location of the search window Is at least 
partially dependent upon the global motion 
vector. 

17. The method of claim 16 wherein the location of 
the search window to be defined in step t) is at 
least partially dependent on the location of the 
video date block within the frame of video data to 
be coded. 
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1 8. The method of daim 1 6 wherein the step of gen- 
erating the global motion vector using a predeter- 
mined method further comprises the step of cal- 
culating the mean of the plural ity of stored motion 
vectors. 

19. The method of daim 16 wherein the step of gen- 
erating the global motion vector using a predeter- 
mined method further comprises the step of de- 
termining the median of the plurality of stored mo- 
tion vectors. 

20. The method of daim 16 wherein the step of gen- 
erating the global motion vector using a predeter- 
mined method further comprises the step of de- 
termining the motion vector occurring with the 
most repetition within the plurality of stored mo- 
tion vectors. 

21. The method of daim 20 wherein the step of gen- 
erating the global motion further comprises the 
step of creating a table of occurrence indices for 
a plurality of possible motion vectors and the step 
of storing the motion vector comprises increment- 
ing an occurrence Index corresponding to the mo- 
tion vector in the table. 



12 



EP 0 652 678 A2 




13 



EP 0 652 678 A2 




CD 

CM 

G 




< 

CM 

52 



EP 0 652 678 A2 



Receive 
DBit 



Aref = F{BPit, GM} 

I ' 



-310 
-312 



Compare Blocks Within 



Receive Data For 
Blocl<s In Argjr 



Best Matched 
Block Within A^ef 

1 



(Position Of First 



^i.dt-BPi.t- Pel In MB, 



I 



ref) 



Transmit MB, 



'ref 



Transmit MVj 



Redefine F, 



ref 




Receive GMV 



GM = GMV 



-332 



FIG. 3A 



15 



EP 0 652 678 A2 




FIG. 3B 



16 



EP 0 652 678 A2 



Receive MVj 




Normalize l\AVj 




4 


^^401 


MVT = IWVT + 
normalized MVj 


i 




Advance counter 
C = C + 1 


402 
'403 


^//Knotherv 
< motion vectorv-?^?^- 



sexpected>< 



More 
blocl<s 
\in frame?/* — -405 



[No 



Determine common 
motion vector 
MVT/C 



Determine global 
motion (= common 
moti on vector) 

z 



Transmit to 
motion compression 
device 



c = o 

MVT = (0.0) 



-^406 



■^407 



^408 



-409 



FIG. 4 



17 



